Skip to content

Shard offline Stim sample files across ranks and reduce pymatching_only counts - #100

Merged
ivanbasov merged 3 commits into
NVIDIA:mainfrom
0z5a:fix/offline-stim-shard-across-ranks
Nov 17, 2026
Merged

ivanbasov merged 3 commits into
NVIDIA:mainfrom
0z5a:fix/offline-stim-shard-across-ranks

Conversation

@0z5a

@0z5a 0z5a commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Offline decoding from .dets files (PREDECODER_STIM_SAMPLES_DIR) reports wrong LER statistics when inference runs with more than one rank. code/scripts/local_run.sh uses every visible GPU and launches torch.distributed.run when there is more than one, and the offline commands in the README do not set GPUS, so this happens on any multi-GPU host unless the user passes GPUS=1 (only offline_smoketest.sh does).

There are two problems:

  1. The LER loop asks each rank for num_samples // world_size shots, but QCDataPipePreDecoder_Memory_from_stim_file always keeps the leading rows of the file. Every rank decodes the same first block. With ising_decoding_pymatching, the all-reduce then counts those shots world_size times and the rest of the file is never decoded. The reported shot count looks right, so nothing hints at the problem, and the standard error is too small by a factor of sqrt(world_size). The syndrome-density reduction path builds the dataset the same way and all-reduces identical shards too; decoder ablation also builds it the same way and reports rank 0's block. Both go through the factory, so they get disjoint shards with this change as well.
  2. pymatching_only returns before the all-reduce, so rank 0 reports only its own shard (num shots = num_samples // world_size).

Repro: d=5, n_rounds=5, O1, default 25-parameter noise model, 262,144 shots per basis written by workflow.task=generate_stim_data, then PREDECODER_DECODE_MODE=pymatching_only WORKFLOW=inference, CPU with the gloo backend (same code path as NCCL):

X LER Z LER
1 process 0.005352 (1403 / 262144) 0.005253 (1377 / 262144)
2 ranks, main 0.005241 0.005203
2 ranks, this PR 0.005352 0.005253

Decoding the same files directly with stim.read_shot_data_file and PyMatching on the circuit's DEM gives 1403 / 1377 errors over the whole file, 687 / 682 on the first half and 716 / 695 on the second half. The 2-rank numbers on main are 687/131072 and 682/131072: rank 0's half only, and rank 1 decoded the same shots. With 4 ranks main reports 0.005203 / 0.005585, the first quarter of the file (341 / 366 errors times 4); with this change 4 ranks also give 0.005352 / 0.005253.

Sharding

The file-backed datapipe applies the global shot limit before splitting the selected rows into contiguous blocks. The first ranks receive one extra shot if needed. Both decode modes reduce their counts across ranks. When decode output is enabled, rank 0 gathers the arrays in file order.

Validation

PYTHONPATH=code python -m pytest -q code/tests/test_offline_stim_decoding.py passed all 25 tests, including the 11-shot file capped at 10 and the two-process Gloo comparison of saved arrays against a single process. The failure-analysis suite passed all 77 tests. Multi-GPU NCCL remains untested.

@copy-pr-bot

copy-pr-bot Bot commented Nov 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@bmhowe23
bmhowe23 requested review from ivanbasov and kvmto and removed request for kvmto November 15, 2026 22:33

@ivanbasov ivanbasov left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution and the detailed reproduction.

The diagnosis is correct: under torch.distributed every rank slices the same leading rows of the .dets file, and pymatching_only returns before the all-reduce. Sharding in the datapipe plus the reduce in pymatching_only addresses both, and single-process behaviour is unchanged (rank=0, world_size=1 reduces to the previous slice). Sharding is preferred over raising for world_size > 1.

Requested changes are inline. One item outside the diff: the README section "Decode the files" should state the multi-GPU behaviour (contiguous per-rank blocks, what happens to remainder shots, and what the PREDECODER_DECODE_OUTPUT_DIR arrays contain).

Comment thread code/evaluation/logical_error_rate.py Resolved
Comment thread code/data/datapipe_stim.py Resolved
Comment thread code/data/datapipe_stim.py Resolved
@0z5a

0z5a commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. All points are in d0f56e8:

  • Output arrays: I went with gathering. Each rank now keeps its rows when PREDECODER_DECODE_OUTPUT_DIR is set, and _gather_decode_rows collects them on rank 0 in rank order (gather_object, which also handles unequal block sizes) before saving. That covers observables, PyMatching predictions, pre-decoder predictions and residual detectors. The two-rank gloo test now writes output dirs and checks that every array matches a single-process run on the same files.
  • Remainder: when num_samples does not cap the shard, the whole file is split and the first file_shots % world_size ranks take one extra shot. If it does cap the shard, every rank takes num_samples // world_size shots from the front and rank 0 prints how many of the file's shots are decoded. The gloo test uses 257 shots on 2 ranks; there is also a 10-shots-on-4-ranks shard test (3, 3, 2, 2).
  • Negative tests: rank out of range (2 and -1 for world_size 2), and fewer shots than ranks. The second check now depends only on the file size and world size, so every rank raises, not just the ones that would get an empty block.
  • _base_cfg is at module level with p_error and meas_basis_test parameters, used by both classes; _distributed_ler_cfg is gone.
  • README "Decode the files" now has a short multi-GPU paragraph: contiguous blocks in rank order, the remainder rule, the capped case, and that the output arrays are gathered in file order.

One note: the old gloo test compared against the matcher in _write_artifact, which at p=0.02 decodes a few shots differently from the pipeline's matcher (a single process shows the same difference). The test now uses a single-process run as the reference.

@ivanbasov

Copy link
Copy Markdown
Member

/ok to test d0f56e8

@ivanbasov ivanbasov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. The follow-up commit addresses all four inline comments and the README. Two remaining items:

  1. _stim_file_shard: when requested_per_rank == file_shots // world_size and the file size is not divisible by world_size, the whole file is decoded, which is more than test.num_samples (11-shot file, num_samples=10, 2 ranks: 11 shots, versus 10 on one rank). Please cap the total at min(test.num_samples, file_shots) so the multi-rank shot count matches the single-rank one in this case as well, and extend the shard tests accordingly.
  2. _gather_decode_rows is exercised on the gloo backend only. Has the PREDECODER_DECODE_OUTPUT_DIR path been run on a multi-GPU host with NCCL? If so, please add the command and result to the PR description.

@ivanbasov

Copy link
Copy Markdown
Member

The gpu-coverage failure on d0f56e8 is unrelated to this change: the branch predates #101, which fixes the cuda-bindings / cuda-core mismatch in the CUDA 12 requirements. Please rebase onto current main together with the remaining changes.

0z5a added 3 commits November 17, 2026 20:11
…ly counts

With PREDECODER_STIM_SAMPLES_DIR set and more than one rank, every rank
built its dataset from the leading rows of the .dets file, so all ranks
decoded the same block and the all-reduce counted it world_size times.
pymatching_only also returned before the all-reduce, so rank 0 reported
only its own shard.

Give each rank a contiguous, disjoint block of the file (rank and
world_size passed through DatapipeFactory), and all-reduce the
pymatching_only error and shot counts like the pre-decoder path does.

Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
With an output directory set, rank 0 wrote only its own block of shots,
so the saved arrays did not cover the shots behind the reported counts.
Gather the observables, PyMatching predictions, pre-decoder predictions
and residual detectors on rank 0 in rank order before saving them.

When num_samples does not cap the shard, split the whole Stim file and
give the first file_shots % world_size ranks one extra shot, so a file
yields the same shot count on N ranks as on one. When it does cap the
shard, rank 0 prints how many of the file's shots are decoded. A file
with fewer shots than ranks is rejected on every rank.

Move the test config helper to module level, add negative tests for the
two ValueError paths and compare a two-rank gloo run against one process,
and document the multi-GPU behaviour in the README.

Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@0z5a
0z5a force-pushed the fix/offline-stim-shard-across-ranks branch from d0f56e8 to 928fb47 Compare November 17, 2026 19:20
@0z5a

0z5a commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main and capped the selected prefix before splitting it across ranks. The offline suite passed, including the 11-shot file capped at 10 and the two-process Gloo output comparison. I have not run the output path with multi-GPU NCCL.

@ivanbasov

Copy link
Copy Markdown
Member

/ok to test 928fb47

@ivanbasov ivanbasov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates. All review comments are addressed and CI is green on 928fb47. The NCCL multi-GPU path for the gathered PREDECODER_DECODE_OUTPUT_DIR arrays is not covered by PR CI and will be checked separately.

@ivanbasov
ivanbasov merged commit 26305d7 into NVIDIA:main Nov 17, 2026
16 checks passed
@ivanbasov

Copy link
Copy Markdown
Member

Merged. Thank you for the contribution and for the quick turnaround on the review comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants